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ABSTRACT 



This paper is a summary of the evaluation of the second 
complete year of the implementation of the Alabama Reading Initiative (ARI) , 
which is designed to achieve 100% literacy by targeting reading performance 
of beginning reading and first-grade students, to expand the reading power 
for second- through twelfth-grade students and to intervene for struggling 
readers at all grade levels. In its second year of implementation, 
approximately 27,700 students, 2,354 teachers, 81 principals, 75 reading 
specialists, 64 higher education partners, and 221 preservice teacher 
education programs were directly involved in the ARI. The evaluation used 
information from student achievement data from surveys completed by teachers, 
reading specialists, principals, higher education partners, and higher 
education reading faculty. Both qualitative and quantitative data were 
analyzed. A pretest-posttest design was used to examine change over time for 
student outcomes. Results and recommendations regarding delivery of the ARI 
are: a large percentage of principals, teachers, reading specialists, and 
higher education partners reported the ARI had substantial positive impacts 
on student literacy; the principal should receive sufficient support, 
direction, and encouragement because their leadership role was related to 
gains in reading scores; the visibility and involvement of the reading 
specialist and higher education partner influenced teachers' attitudes toward 
the helpfulness of these individuals; higher education faculty in the program 
are often confronted with inadequate or competing reward structures; teachers 
involved in the first 2 years of the program felt that continued professional 
development would help sustain their enthusiasm; the Stanford Achievement 
Test, Level 9 is not the most appropriate measure for monitoring progress 
toward program goals; tests currently used to assess early literacy skills of 
kindergarten through second-grade students are inappropriate for use as tools 
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for evaluating gains associated with ARI ; and case studies of particularly 
effective schools should be conducted. (RS) 
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EXECUTIVE SUMMARY 



This is a summary of the evaluation of the second complete year of 
implementation of the Alabama Reading Initiative (ARI). In its first year of 
implementation, evaluators from the University of Alabama at Birmingham (UAB) and 
the University of Alabama at Huntsville (UAH) found that students in ARI Literacy 
Demonstration Sites (LDSs) demonstrated greater gains on the Stanford Achievement 
Test, Ninth Edition (Stanford 9) than would have been expected without the benefit of the 
ARI. These students also scored better than students in comparison schools on the 
Stanford 9. Included in this second evaluation report are the 16 original LDSs (called 
Group A) that began implementation during the 1998-99 school year and the 65 new 
LDSs (called Group B) added during the 1999-2000 school year. 

The Alabama Reading Initiative 

The Alabama Reading Initiative (ARI) is a statewide effort directed by the State 
Department of Education to improve reading instruction and achieve 100% literacy 
among K-12 public school students. Unique features of the ARI include: 

• Participating schools must set a goal of 100% literacy; 

• At least 85% of the faculty must attend a 10-day professional development 
institute; 

• The principal must attend the institute and lead approximately 10 hours of faculty 
meetings during which faculty members develop implementation plans; 

• Faculties must be willing to adjust reading instruction to reflect research-based 
practices; 

1 See “Evaluation of the Alabama Reading Initiative 1998-1999,” prepared for the Alabama Commission 
on Higher Education by the Center for Educational Accountability, University of Alabama in Birmingham, 
Birmingham, AL. 
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• Attention is paid to increased performance of all students: beginning readers, 

accomplished readers, and struggling readers; and 

• LDSs fonn partnerships with educators from teacher education programs in 

Alabama’s Institutions of Higher Education (IHEs). 

In its second year of implementation, approximately 27,700 students, 2,354 
teachers, 81 principals, 75 reading specialists, 64 higher education partners to those 
schools, and 21 pre-service teacher education programs in Alabama’s IHEs were directly 
involved in the ARI. 

The Evaluation Plan 

The second year evaluation of the ARI was designed to document progress toward 
100% literacy in the 81 LDSs, identify factors associated with successful results, and to 
ascertain the impact of ARI on pre-service teacher education as a result of involvement in 
the ARI. The evaluation answers six questions: 

1 . To what extent are ARI schools making progress toward 100% literacy? 

2. Which ARI schools are making progress toward 100% literacy and which are 
not? 

3. What factors are related to school outcomes? 

4. Why are some ARI schools making more progress than others? 

5. To what extent are the elements of ARI reflected in pre-service teacher 
education programs throughout Alabama? 

6. What ARI factors are related to change in pre-service teacher education 
programs? 
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These questions were addressed using information from the Stanford Achievement Test, 
Ninth Edition (Stanford 9), and surveys completed by teachers, reading specialists, 
principals, higher education partners, and higher education reading faculty. 

Key Findings 

Answer to Question 1: ARI schools made more progress toward 100% 
literacy than did non-ARI schools. Specifically: 

• Improvements in Normal Curve Equivalency (NCE) scores across Reading 
Comprehension, Reading Vocabulary, and Total Reading between 1998 and 2000 
averaged 1.05 for Group A ARI schools and .24 for non-ARI schools (all other 
Alabama public schools). Improvements in NCE scores across all these reading 
subtests between 1999 and 2000 averaged .28 for Group B ARI schools and .20 for 
non-ARI schools. These improvements translated into small but positive differences 
in effect sizes 3 favoring ARI schools for each reading subtest for Group A and B. 

• Asa group, ARI schools decreased the population of “struggling readers” (students 
scoring in the bottom three stanines on the Stanford 9 reading subtests) by as much 
as 10%. Parallel reductions in the percentage of struggling readers in non-ARI 
schools over the same periods of time were less than half that of the ARI schools: 



Decreases in the Population of Struggling Readers 
Percentage of Decrease Percentage of Decrease 

From 1998-2000 From 1999-2000 



Subtest 


Group A 


Non-ARI 


Group B 


Non -ARI 


Vocabulary 


10.19% 


4.68% 


6.39% 


3.01% 


Comprehension 


8.45% 


.02% 


6.56% 


1.42% 



2 An NCE score is similar to a percentile score in that it can take values between 1 and 99. Unlike 
percentile scores, NCE scores can be averaged, allowing means to be calculated across schools and across 
grade levels. 

3 Effect sizes are indices of the practical significance of differences between average scores. They are 
determined by computing the differences between means for two groups and then dividing the difference 
by the amount of dispersion in the scores (standard deviation). Effect sizes may be positive or negative. 
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• ARI schools increased the percentage of “grade-level readers” (students scoring in the 
top five stanines) more than non-ARI schools over the same periods of time. ARI 
Group A schools increased at rates that ranged from 2.18% to 4.56% as compared to 
rates of .72% to 2.31% for non-ARI schools. The rate of increase for Group B 
schools ranged from 1.28% to 1.84% as compared to rates of increase for non-ARI 
schools that ranged from .87% to 1.01%. 

Answer to Question 2: Findings support that the vast majority of ARI schools 
are making progress toward the goal of 100% literacy. Still, there is considerable 
variability in the gains made by ARI schools. 

Approximately 70 percent of ARI schools demonstrated small gains (effect sizes 
greater than 0 and less than .40) on Reading Vocabulary, Reading Comprehension, or 
Total Reading; and an additional five percent of ARI schools made moderate (effect sizes 
between .40 and .70) or large gains (effect sizes of greater than .70) on these measures. 

An example of the trends in performance across schools is seen in Figure 1 on the next 
page. This figure graphically displays the range of change in the percentage of 
“grade-level readers” (i.e., students scoring at or above stanine 5 on the Reading 
Comprehension subtest) for Group A and Group B. The figure reveals that while the 
majority of schools in Cohorts A and B made positive changes, some schools do 
particularly well and some schools do less well in terms of the indicator. Similar ranges 
of perfonnance between schools were found for all indicators of literacy. 
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Figure 1. Change in Percentage of Students in Group A and Group B Schools 
Scoring at or above Stanine 5 on the 
Stanford 9 Reading Comprehension Subtest 
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Because of the variability illustrated in the figure above, it was important to 
identify factors that might account for differences in the performance of ARI schools. 
Efforts to identify such factors are summarized in Questions 3 and 4 later in this 
Executive Summary. 

Other indicators of positive, yet variable progress among ARI schools include the 
following: 

• Stanford 9 scores revealed positive progress toward 100% literacy on one or more 
of the reading subtests (effect sizes ranging from .01 to .72) in 13 of the 15 Group A 
schools that have Stanford 9 scores and 45 of the 61 Group B schools. 

• Decreases in the percentage of struggling readers (.02% to 18%) occurred in 10 of 
the 15 Group A schools and 47 of the 61 Group B schools on one or more of the 
reading subtests. 

• Increases in the percentage of students scoring “on grade level” (.05% to 28%) 
occurred in 13 of the 15 Group A schools and 51 of the 61 Group B schools on one or 
more of the reading subtests. 
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• Systematic and substantial decreases in discipline and special education referrals 
were found in the subset of ARI schools (7 in Group A and 27 in Group B) that 
reported such data. Discipline referrals in the reporting Group A schools decreased 
by 67%, from 1,795 referrals in 1998 to 596 referrals in 2000. The average decrease 
in discipline referrals across the 27 reporting Group B schools was 23%. 

• Asa group, the eight reporting Group A schools decreased special education referrals 
by 28% from 1998 to 2000. The 33 reporting Group B schools decreased special 
education referrals by 14% from 1999 to 2000. 

To understand why some schools demonstrated marked improvement and others 
showed losses in achievement, subsets of higher- and lower- performing schools were 
identified. The method used for identifying higher- and lower- achieving ARI schools 
included approximately 26 improvement indicators from the Stanford 9. Schools 
included in the final subsets of higher- and lower-performing ARI schools were those that 
demonstrated substantial consistency across these improvement indicators. Seven Group 
A schools were identified as higher-performing schools and two Group A schools were 
identified as lower-performing. In Group B, 14 schools were identified as higher- 
performing and eight were identified as lower-performing 

Answer to Questions 3 and 4: Several factors discriminated higher- and lower- 
performing ARI schools. The leadership of the principal and the helpfulness of the 
reading specialist had the greatest impact on student achievement. Specifically: 

• The Principal as ARI Champion . Other than the ARI training itself, data point to 
the principal as having the greatest impact on student achievement. Correlations 4 
between principal leadership scores as provided by teachers, higher education 
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partners, and reading specialists and Stanford 9 effect were .20 or greater between 
leadership and one or more of the reading scores. In higher-performing ARI 
schools, the principal supported implementation by facilitating ongoing 
professional development; providing resources and materials; adjusting schedules 
in order to enhance reading instruction; supporting and monitoring teacher 
implementation in the classroom; and encouraging faculty members. In lower- 
performing schools little or no support was noted. 

• The Hands-On. Helpful Reading Specialist . The reading specialist also made a 

difference. The large majority of teachers in both higher-performing and lower- 

performing schools considered the role of the reading specialist as important to 

the successful implementation of ARI. Seventy percent (70%) of the teachers 

expressed in surveys that the reading specialist was important to serving the needs 

of students and teachers. However, teacher and principal ratings of the helpfulness 

of the reading specialist were greater in higher-performing schools than in lower- 

performing schools. Correlations in excess of .20 were found between teachers’ 

ratings of the helpfulness of the reading specialist and the scores for Reading 

Comprehension and Reading Total on the Stanford 9. Teachers reported that the 

reading specialist impacted reading skills of struggling readers by working with 

them directly and by encouraging and assisting teachers. Teachers also 

commented on the help that reading specialists provided through program 

oversight, professional development, and materials and resources. 

4 Correlations are statistical values that range from -1.0 to 1.0. They relate one score to another score. For 
example, if scores on principal leadership are positively correlated with scores on the Stanford 9, we can 
conclude that increases in principal leadership are associated with gains on the Stanford 9. The higher the 
correlation is, the greater is the association between leadership and gains. 
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• The Deeply Involved Higher Education Partner . Differences existed in higher- 
and lower-performing schools in the perceived helpfulness of the higher education 
partner, particularly as evaluated by the reading specialist. In higher-performing 
schools, reading specialists report that higher education partners had direct 
involvement with students, teachers, and reading specialists. They conducted 
professional development sessions; modeled instruction; arranged for workshops; 
worked with struggling readers; worked one-on-one with teachers in their 
classrooms; trained new teachers in the ARI modules; and consulted with the 
principal. Higher education partners in lower- performing schools were reported 
to serve more as an emotional support. They listened and encouraged but were 
less likely to be reported as offering subject matter-related expertise, 
demonstrations, consultation, and problem-solving. 

• Other Implementation Influences . Other factors discriminating higher- and lower- 
performing schools were teacher reports of the rates at which ARI components 
were implemented, ongoing professional development hours, increased attention 
to student reading time, greater attention to reading instructional strategies, more 
ongoing assessment of student progress, and increased focus on struggling 
readers. 

In addition, teachers responding to survey data reported that ARI had positive 
impacts on their teaching and student learning. Changes in teaching included use of 
research-based strategies, increases in time dedicated to student reading, increased 
confidence and enthusiasm toward reading instruction, increased awareness of 
struggling readers, and positive school-wide change in the learning environment. 
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Changes reported in student learning included improved reading skill, improved 
writing abilities, and increased motivation to read. 

Answer to Questions 5 and 6: Findings indicate that changes in course content 
occurred in pre-service teacher education programs throughout Alabama as a result 
of the ARI. Survey data suggest further that teacher education faculty members 
perceived the ARI to benefit their teaching and their pre-service students. 
Specifically: 

• All higher education partners and reading faculty who responded to surveys indicated 
changes in course content that reflected material contained in the ARI teacher training 
modules and other ARI-published documents. 

• All higher education partners and reading faculty who responded to surveys indicated 
that current course syllabi incorporated the new reading standards adopted by the 
Alabama State Board of Education in December 1999. Those standards reference 
specifically Knowledge and Skills Teachers Need to Deliver Effective Reading 
Instruction, a document developed by the ARI and published by the Alabama State 
Department of Education in February of 1998. 

• A frequently mentioned impact of the ARI on pre-service teacher education programs 
was the effectiveness of pre-service teaching experiences in Literacy Demonstration 
Sites where students could observe, complete field experiences and internships, and 
become a part of research-based, effective practice. 

Limitations to the Evaluation 

The results of the Year Two Evaluation of the Alabama Reading Initiative must 
be interpreted within the context of several conditions: 
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1 . Student outcome data came almost exclusively from the Stanford Achievement Test - 
Ninth Edition (Stanford 9), a norm-referenced measure that forms the accountability 
system in Alabama. Three items are worth considering when interpreting results: 

• The primary purpose of a norm-referenced instrument is to compare achievement 
of individuals or groups to that of the norming group. Norm-referenced 
achievement instruments such as the Stanford 9 are not created as direct measures 
of the literacy level of the reader. 

• Since the Stanford 9 is the accountability instrument for Alabama, the evaluation 
of the ARI is occurring within a competitive environment where all schools in 
Alabama are expected to perform to a state-established standard on this test. The 
fact that ARI schools made greater gains than those found in non-ARI schools is 
noteworthy given the efforts that all Alabama schools are making to improve 
achievement on the Stanford 9. 

• Given the nature of norm-referenced tests, factors such as regression toward the 
mean and insensitivity to small differences or gains could account for some 
variability in gains reported for ARI schools. 

2. The tests currently used to assess early literacy skills of students in kindergarten 
through second grade are not designed to assess ongoing development of reading 
ability during the first three years in school. Therefore, the evaluation does not 
present information concerning the progress of students in Grades K-2, where 
progress might be expected to occur at a greater rate than in upper grades. 

3. All information from survey data was self-reported and was not verified by on-site 
observations. 
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4. A specific model was used to identify higher-performing and lower-performing 

schools. It is likely that other models using different selection criteria would, to some 
degree, identify different subsets of higher- and lower-performing schools. 

Therefore, conclusions regarding differences between higher-performing and lower- 
performing schools should not be generalized beyond the model used for this 
evaluation. 

Recommendations 

Based on the findings of the evaluation as well as the limitations discussed above, 
the evaluators make the following recommendations to the ARI and subsequent 
evaluations of the initiative: 

1 . Place greater emphasis on the central role of the principal in facilitating the 
effective implementation of ARI. 

2. Work to increase the helpfulness of all reading specialists. Additionally, seek 
ways of increasing the number of reading specialists, since a prerequisite to 
“helpfulness” appears to be their availability to work directly with struggling readers 
and with teachers. This is particularly important since the number of ARI schools is 
growing substantially (425 in 2001-2002), but the rate of state funding is not 
increasing proportionately. 

3. Develop strategies to increase the direct involvement of higher education 
partners in assisting classroom teachers. This is challenging since the number of 
ARI schools will grow to 425 in 2001-2002, and there are relatively few higher 
education reading faculty who could provide direct assistance to classroom teachers. 
Additionally, there is a need for creative incentives that reward service to ARI 




schools within the traditional higher education faculty evaluation framework of 
service, scholarship, and teaching. 

4. Explore ways of providing continued professional development, support, and 
recognition to schools beyond the first year of implementation. 

5. Increase, to the extent possible, the use of criterion-referenced measures that are 
designed to document progress of students toward reading at or above grade 
level. Implementing fall and spring testing using the individual or group fomi of the 
criterion-referenced reading assessment currently administered in kindergarten 
through Grade 2 and/or linking these measures psychometrically would permit the 
use of available instruments for this purpose. 

6. Include longitudinal data on “matched” students across grades within schools in 
future evaluation efforts. 

7. Include case studies of the higher-achieving and lower-achieving ARI schools 

that include observation data as well as interview data. 

8. Require the collection of other school outcome measures such as discipline 
referrals, special education referrals, and library circulation so that such data can be 
reported by all schools participating in the ARI. 
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